1 Statistical Models

1.1 Parametric vs Nonparametric Models

Parametric models are distributions indexed by ${\displaystyle \theta \in \mathrm{\Theta }}$. So ${\displaystyle \mathcal{P}=\{P_{\theta }|\theta \in \mathrm{\Theta }\},\mathrm{\Theta }\in {\mathbb{R}}^d}$. (${\displaystyle d}$ is called model dimension)
Denote ${\displaystyle {\mathbb{P}}_{\theta }(\cdot ),{\mathbb{E}}_{\theta }(\cdot )}$, also indexed by ${\displaystyle \theta }$.
On the contrary, non-parametric models means no natural way to parameterize ${\displaystyle \mathcal{P}}$ by real vector.

Simple case for non-parametric models is ${\displaystyle X_1,\cdots ,X_n\stackrel{\mathrm{i}.\mathrm{i}.\mathrm{d}}{\sim }P}$, ${\displaystyle P}$ is ANY distribution on ${\displaystyle \mathbb{R}}$. Then ${\displaystyle \mathcal{P}=\{P^n|P\text{ is a distribution on }\mathbb{R}\}}$, ${\displaystyle X=(X_1,\cdots ,X_n)\sim P^n}$.

However, we can use "parametric notation" ${\displaystyle \mathcal{P}=\{P_{\theta }|\theta \in \mathrm{\Theta }\}}$ WLOG. (we can always denote ${\displaystyle \theta =P,\mathrm{\Theta }=\mathcal{P}}$)

1.2 Bayesian vs Frequentist Inference

So far we assume data ${\displaystyle X}$ follows a distribution ${\displaystyle P_{\theta }}$ for parameter ${\displaystyle \theta }$ we want to determine. Sometimes we will add the Bayes assumption that ${\displaystyle \theta }$ itself is random, drawn from a known distribution ${\displaystyle \mathrm{\Lambda }}$ we call prior.
This helps reduce the problem of inference, and we can focus on the conditional distribution of ${\displaystyle \theta |X}$.
However, before introducing Bayesian Inference, we assume ${\displaystyle \theta }$ as a fixed value in ${\displaystyle \mathrm{\Theta }}$.

2 Estimation

We want to determine the value of a parameter in a parametric model.

• Skeptical answer: could be anything.
• Bayesian answer: assume ${\displaystyle \theta }$ random with prior distribution, and calculate conditional distribution of ${\displaystyle \theta |X}$.
• Frequentist answer: inductive behavior: find a method for using ${\displaystyle X}$ to estimate ${\displaystyle \theta }$. E.g. ${\displaystyle {\delta }_0(X)=\frac{X}{n}}$. Show it generally works well for every ${\displaystyle \theta }$.
  General step: ${\displaystyle \mathcal{P}=\{P_{\theta }|\theta \in \mathrm{\Theta }\}}$. Estimate ${\displaystyle g(\theta )}$. Observe ${\displaystyle X}$, calculate estimate ${\displaystyle \delta (X)}$ using ${\displaystyle \delta (\cdot )}$.

In brief, we have two primary strategies to choose an estimator:

1. Summarize the risk function by a scalar.
2. Restrict attention to a smaller class of estimators.

2.1 Comparing Estimators

2.2 Resolving Ambiguity

There is no estimator that uniformly attains the smallest risk among all estimators. Like the brute ${\displaystyle \delta (X)\equiv \frac{1}{2}}$, it performs the best when ${\displaystyle \theta }$ is actually ${\displaystyle \frac{1}{2}}$. There are two main strategies to resolve this ambiguity.

Summarize the risk function by a scalar.

• Average-case risk (Bayes estimation): minimize some (weighted) average of the risk function over ${\displaystyle \mathrm{\Theta }}$: $$\underset{\delta (\cdot )}{min}{\int }_{\theta }\mathcal{R}(\theta ;\delta )\mathrm{d}\mathrm{\Lambda }(\theta ).$$
  If ${\displaystyle \mathrm{\Lambda }(\mathrm{\Theta })<\mathrm{\infty }}$, we can assume WLOG that ${\displaystyle \mathrm{\Lambda }}$ is a probability measure. Then this average is the same as ${\displaystyle {\mathbb{E}}_{\theta \sim \mathrm{\Lambda }}[\mathcal{R}(\theta ;\delta )]}$ called the Bayes risk. An estimator minimizing the Bayes risk is called the Bayes estimator.
• Worst-case risk (Minimax estimation): $$\underset{\delta (\cdot )}{min}\underset{\theta \in \mathrm{\Theta }}{sup}\mathcal{R}(\theta ;\delta ).$$ This will push us to choose estimators with flat risk functions, like ${\displaystyle {\delta }_2(X)=\frac{X_2}{n+4}}$ from the example.

Restrict the choice of estimators
Unbiased estimation: we can demand an estimator to satisfy ${\displaystyle {\mathbb{E}}_{\theta }[{\delta }_0(X)]=g(\theta ),\mathrm{\forall }\theta \in \mathrm{\Theta }}$.
Under unbiasedness, we can clearly define the optimal estimator called UMVU estimator. From the above example, ${\displaystyle {\delta }_0(X)=\frac{X}{n}}$ is actually the UMVU estimator.